For several years, the CCD and CMOS technologies have been used to equip the sensor of digital still or animated imagery devices. Initially more performing, the CCD sensors have gradually been supplanted by the CMOS sensors that present a superior quality/price ratio. Nevertheless, the inferior cost of manufacturing of the CMOS sensors can be explained due to technological compromises of which Rolling Shutter is a part and is employed on a wide range of movie cameras, starting from the sensors integrated in the mobile phones to the high end camcorders.
More precisely, today a digital imagery sensor (CCD or CMOS) is made of a 2D arrangement (commonly called matrix) of photosensitive elements (or pixels), usually according to a rectangular grid. Every photosensitive element converts the intensity of the light that it receives into an electric voltage. When a new picture is acquired, it is necessary to read all of these electric voltages and to transfer them, through a bus, towards the processing unit and/or a storage support. This transfer is carried out in a linear manner, that is, pixel by pixel, after having concatenated all the lines that make up the picture.
The behavior of the CCD and CMOS is different during this transfer:                At a precise moment the CCD sensors freeze the values of the voltages for all pixels and store them in a buffer memory. The voltages sent alongside the data bus correspond therefore to the sensor state at a given moment;        Generally the CMOS sensors do not contain any memory buffer. The voltages transmitted alongside the data bus correspond to the state of every pixel at the time of the reception of the sync pulse that is slightly shifted in comparison with its neighbors in order to avoid the collision between two values of different origin on the bus.        
This behavior is at the origin of the expression of rolling shutter. In fact, if one supposes that the reading of a line of the picture is almost instantaneous, the final picture is constituted of an assembly of lines measured at slightly different instants. This is equivalent, in terms of pictures obtained, to have a mechanical shutter individually exposing every line and rolling from down to up.
This process is similar to that used on cameras (film and digital) equipped with focal plane shutter to expose sensitive surface for a period less than the time required for vertical displacement of the curtain: the rise of the first curtain (which gives insulation of the film) is almost immediately followed by the rise of the second curtain (which stops the sunlight) so that at every moment light reaches the band located between the 2 curtains of the shutter. This sliding window therefore exposes each strip of film during a specified short time that would have been impossible to achieve if we waited that the first curtain finish its shift before starting the second1. 1 One can observe this sliding window if one uses a flash while the speed of closing of the case is superior to its said synchro-X value: the short light of the flash fixes the position of the exposure film at the time of triggering
Types of Artifacts Encountered
From the description of the process of the rolling shutter, two types of artifacts can be expected:    1. Geometric distortions: if the fixed camera acquires a mobile object, then this object will be moved between the acquisitions of two successive lines of the image. Its appearance will then be deformed: if for example it is a rectangle moving horizontally, each line of the rectangle is slightly shifted versus the previous one to form a parallelogram (see FIG. 1): the time difference at the level of the sensor is translated into a spatial difference from each line of the object observed. This reasoning is obviously valid if the scene is fix and the camera mobile;    2. Brutal differences of radiometry: if the scene contains a flashing lamp (lighthouse at sea, emergency rotating light . . . ) and that this lamp goes off during the acquisition of the scene, then a part of the image has received much lesser light than the other because of the disappearance of this source of light. The integration time of light on each pixel is fixed for the entire image; the exposure calculated including the lighted lamp will immerse a portion of the image in the dark. For example, one portion of the facade of a building is properly shown while another portion will be underexposed and very dark.
Both types of defects are found in practice. When researching on this subject, one finds that the geometric imperfections are sometimes divided into two categories, namely the skew effect and jellocam effect. It is in fact the same phenomenon induced by the rolling shutter, but with a greater or less pronounced degree. The first term refers to the less serious cases, namely geometric defect presented above: a rectangle is transformed into a parallelogram by a shift or shear. The second term refers to more serious cases, which occur when the motion of the camera (for a fixed scene) changes faster than the camera acquires an image. The different parts of the image seem animated with their own movement, as if the observed scene was wobbling, like a jelly dessert. This case occurs relatively frequently when the camera is subjected to vibration of generally high frequency.
State of the Art Solutions
Computer Vision: We can trace the first analysis of the impact of rolling shutter in 2005 [GMS05], accompanied by a modeling of the associated geometric defects and a method to correct them. Just as [NFM06] (which is inspired from the aforementioned), these works of Geyer, Meingast and Sastry fall within the specific context of mobile robotics. The main motivation of the authors is to use simple and low cost sensors (here, web cams connected via Firewire) for odometer tasks and environmental perception of the robot. The rolling shutter indeed disrupts the traditional methods of 3D from the video because it distorts the objects observed during the movements of the robot, thus introducing additional noise in the measures taken from the images.
The authors develop a rigorous geometric modeling of the rolling shutter and the associated optical flow equations to remove the errors induced by the CMOS in the perception calculations. Nevertheless, the proposed methods are used to correct images acquired by the camera only in case of full knowledge of the movement of the camera and the 3D depth of the points observed, which is not practical for generic imaging applications (and is already a strong constraint for the robotics in a controlled environment). Finally, they are also the first to introduce the hypothesis of instantaneous acquisition of a line (i.e. all pixels of the same image line are measured simultaneously and the only time lag is between 2 different lines) which will then be repeatedly used by all other authors.
General public imagers: The interest of researchers for the correction of rolling shutter in consumer applications is also quite limited since only 2 teams have published works on this subject. The first is a South Korean team from POSTECH in 2006 [IKH06] with extended work in [hCH07, CH08]. The purpose of their work is ultimately to correct and stabilize the videos from general purpose sensors rather than measuring positions in 3D. In comparison with [GMS05] they remove the 3D position of the points of the image of parameters to know or to estimate.
Instead, they assume (classic stabilization of videos) that transformation between two successive frames of a video stream can be represented by an affine transformation (6 degrees of freedom) or by a homography (8 degrees of freedom), knowing that proven methods can estimate these changes. They then write about the analytic expression of the transformation chosen with and without rolling shutter (under the hypothesis of instantaneous reading of a line and flat scene, the rolling shutter transforms a homography into another homography) and different types of camera movement (translation, zoom). Then, by identifying words, they separate the apparent motions measured in the images into a component due to rolling shutter and another due to the movement chosen by the operator (traveling, zooming) on which they also carry out calculations of stabilization. Their method is more close to the concerns of an amateur filmmaker than [GMS05] and gives good results for the cases treated. However, the panel of cases processed is limited to simple movements of the camera (translation, zoom) because of the complexity of expressions and calculation inaccuracies.
The second team having published results on the subject of the correction of the effects of rolling shutter in consumer applications is a team in Taiwan [CLC08]. This team does not seem to have known the results obtained by the team POSTECH (the opposite not being true). Although more recent, their approach seems a step backwards because it is less generic than the previous one: it only supports translations as camera movements. Moreover, most of the effort seems to have been put in producing a visually plausible image interpolation process rather than in the skew estimation. The principle of their method is as follows:    1. An optical flow algorithm used to estimate the apparent motion (2D translation) of image blocks;    2. The domain of movements of different blocks is averaged to estimate the translation of the image center, which serves as an estimator of global movement of the scene;    3. The overall movement of the image is regarded as unchanged by the rolling shutter,    4. The camera path over time is obtained by polynomial interpolation (degree 3) between the successive positions of the center of the image;    5. The lines of the image are finally transformed with respect to the interpolated path for the rectified images.
It may be noted also that none of the methods presented in this paragraph is able to handle the case of moving objects within the scene and that all the methods mentioned will have difficulties to handle jellocam cases. Indeed, in this case the global movement of the image is poorly defined and the hypothesis of global rigid motion (even obtained by interpolation) is no longer valid.
It is also interesting to note that:    1. Unlike existing methods, our method of correction of geometric artifacts is theoretically capable of handling both moderate (skew effect) and large (jellocam effect) defects, because there are no hypothesis about the relative movements of the scene and camera. In practice, methods for estimating the rolling shutter with varying degrees of freedom (and therefore more or less robust) can be used depending on the severity of defects we wish to correct;    2. The rolling shutter artifacts have long been known in the case of scanning sensors (of size one or several pixels disposed on a single line), where a sensor is moved in the focal plane of the lens to create the final 2D image. However, parades applied previously are inapplicable here, since they consist in providing heavy rigid supports support to the camera to absorb the vibrations and . . . to replace the linear acquisition sensor by a 2D matrix.